Calibration, customization, and improved user experience for bionic lenses

ABSTRACT

The present disclosure relates to calibration, customization, and improved user experiences for smart or bionic lenses that are worn by a user. The calibration techniques include detecting and correcting distortion of a display of the bionic lenses, as well as distortion due to characteristics of the lens or eyes of the user. The customization techniques include utilizing the bionic lenses to detect eye characteristics that can be used to improve insertion of the bionic lenses, track health over time, and provide user alerts. The user experiences include interactive environments and animation techniques that are improved via the bionic lenses.

CROSS-REFERENCE TO RELATED APPLICATIONS

This is a divisional of U.S. patent application Ser. No. 16/455,012,filed Jun. 27, 2019 and titled “CALIBRATION, CUSTOMIZATION, AND IMPROVEDUSER EXPERIENCE FOR BIONIC LENSES,” which is hereby incorporated hereinby reference in its entirety for all purposes.

FIELD

The present disclosure relates generally to bionic contact lenses wornon a user's eye.

BACKGROUND

Bionic contact lenses, such as a smart contact lenses, that are worn orinserted into the eyes of users are quickly developing. Generally, theselenses may include circuitry and sensors that provide or generateinformation that can be displayed direct into the user's eye. Forexample, some bionic contact lenses include displays that generateimages presented directly into a user's eye. Some bionic lenses alsoinclude integrated cameras that capture images from approximately thesame viewpoint as the user. Many technology fields, such as alternatereality and virtual reality technologies, are looking to leverage thelenses for new applications and techniques. For example, with AR/VRapplications, the display on the bionic lens generates images that canbe directly overlaid with the user's “real world” vision, since theimage formed by the light from the display is projected, along withlight from the world, onto the user's retinas, forming the user's view.However, to operate accurately and provide an immersive user experience,the calibration and accuracy of the bionic lenses is important, currentlenses may not be accurately calibrated, hindering the user experience.

Additionally, certain aspects of the lenses can be leveraged to furtherincrease the user immersion, as well as provide important and usefultools for the applications utilizing bionic lenses.

SUMMARY

In one embodiment, a system to calibrate bionic lenses configured to bepositioned on the eye of a user is disclosed. The system includes areplica human eye model to receive a bionic lens, the model including animage sensor positioned at a focal length corresponding to an averagehuman focal length, a display that displays a calibration pattern, thedisplay being in optical communication with the image sensor and thebionic lens, and a computer in electrical communication with the imagessensor. The image sensor captures at least one calibration imagecorresponding to the displayed calibration pattern, the at least onecalibration image corresponding to the calibration pattern as viewedthrough the bionic lens and the computer compares the captured at leastone calibration image to the calibration pattern as displayed, or fitsthe captured calibration image to a mathematical model, and determinesthe characteristics or intrinsic parameters (e.g. focal length,decentering, distortion) of the bionic lens. Alternatively, the rearsurface of the model can be a diffuse screen, and an external cameracaptures an image of the calibration patterns projected through thebionic contact lens and model eye.

In another embodiment, a method to calibration distortion of a bioniclens is disclosed. The method includes displaying by an onboard displayof the bionic lens a calibration pattern into an eye of the user wearingthe bionic lens; capturing by an image sensor a calibration imagecorresponding to light exiting the eye generated by the displayedcalibration pattern reflecting from a retina of the eye; comparing bythe processor the calibration image to the calibration pattern todistortion introduced by the bionic lens and optionally the human orbiological lens; and generating by the processor a compensation map forthe bionic lens and optionally the human or biological lens tocompensate for the distortion.

In one embodiment, a method to determine eye characteristics of a userwearing one or more bionic lenses is disclosed. The method includescapturing by a first bionic lens worn in a first eye of the user a firstimage corresponding to a calibration pattern positioned at a firstorientation of the user relative to the calibration pattern, capturingby the first bionic lens a second image corresponding to the calibrationpattern positioned at a second orientation of the user relative to thecalibration pattern; and analyzing the first image and the second imageby a processor to determine the eye characteristics of the user. Thismethod can be used to determine interpupillary distance, calibrate theonboard camera of the bionic lens, and/or determine pupil swim. Itshould be noted that in some embodiments, multiple images are capturedfrom various orientations as needed, such as n images may be capturedand analyzed.

In another embodiment, a method to determine an interpupillary distancebetween two eyes is disclosed. The method includes capturing by a bioniclens camera an image corresponding to a reflection of two eyes on asurface, analyzing the image by a processor to detect an iris locationfor the two eyes; and estimating by the processor the interpupillarydistance based on the detected iris location for the two eyes. In oneembodiment, the irises are detected and the iris diameter can be used asa reference scale to determine the interpupillary distance.

In yet another embodiment, a system for determining physicalcharacteristics of an object is disclosed. The system includes a contactlens including an image sensor and a processor in communication with thecontact lens. The display emits a calibration pattern that reflects onthe object, the image sensor captures a calibration image of thecalibration pattern as reflected on the object, and the processorcompares the calibration pattern to the captured calibration image toanalyze distortions of the calibration pattern and determine one or moreobject characteristics.

In an embodiment, a method to determine a shape of the object isdisclosed. The method includes projecting light from a first bionic lenspositioned on a first eye of a user onto the object, capturing a firstimage corresponding to the light projected by the first bionic lens ontothe object, capturing a second image corresponding to the lightprojected by the second bionic lens onto the object, projecting lightfrom an external source positioned on the user onto the object, andanalyzing the first image and a first location corresponding to thefirst eye, the second image and a second location corresponding to thesecond eye, and a third image and a third location corresponding to theexternal source to determine the shape of the object.

In another embodiment, a method to determine vergence and focusinformation for a user is disclosed. The method includes activating afirst light display of a first bionic lens positioned on a first eye ofthe user to emit a first beam of light, activating a second lightdisplay of a second bionic lens positioned on a second eye of the userto emit a second beam of light, detecting a first location correspondingto a reflection point of the first beam of light on an object, detectinga second location corresponding to a reflection point of the second beamof light on an object, analyzing the first location and the secondlocation to determine an intersection location of the first beam oflight and the second beam of light, and utilizing the intersectionlocation to determine a vergence and focus area of the user.

In one embodiment, a method for converting eye movements of an actorinto computer animated movements is disclosed. The method includesactivating a first bionic lens worn on a first eye of the actor to emita first light, activating a second bionic lens worn on a second eye ofthe actor to emit a second light, tracking movement of the first lightand the second light; and converting the tracked movements of the firstlight and the second light into animated movement of a first charactereye and a second character eye, respectively.

In yet another embodiment, a method to generate biometric informationfor a person is disclosed. The method includes projecting a lightpattern by a display positioned on a contact lens into an eye of theperson, capturing by an image sensor positioned on the contact lens, aplurality of images of the eye corresponding to a display sequence ofthe light pattern; generating an eye map (e.g. retinal image)corresponding to the captured plurality of images, wherein the eye mapis specific to the eye of the person.

In another embodiment, an interactive environment is disclosed. Theenvironment includes a plurality of smart objects including a camera anda computer in communication with one another and a bionic lens systemconfigured to be worn by a person within the interactive environment,including a first lens having a first display and a second lens having asecond display. The first display and the second display emit a firstlight pattern and a second light pattern and the plurality of smartobjects detect the first light pattern or the second light pattern,analyzing the first and second light patterns to generate a customizedexperience specific to the person.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a front elevation view of a bionic lens system positioned onan eye.

FIG. 1B is an exploded partial cross-section view of the system of FIG.1A.

FIG. 1C illustrates an implementation of the system where the bioniclens includes a retinal facing image sensor and outward facingprojector.

FIG. 1D illustrates an implementation where the bionic lens is used asan in-to-out projection system.

FIG. 1E illustrates an implementation where the bionic lens is used as aretinal scanner.

FIG. 1F illustrate an implementation where the bionic lens utilizes thedisplay as a light sensor, capturing light from the environment asreflected from the retina.

FIG. 2 is a simplified block diagram of the bionic lens system of FIG.1A.

FIG. 3A is a cross-sectional ray diagram of the bionic lens system beingutilized as an eye as a camera system.

FIG. 3B is a cross-sectional ray diagram of the bionic lens system beingutilized as an eye as a projector system.

FIG. 3C is a cross-sectional ray diagram of the bionic lens system beingutilized as a projector-cam system.

FIG. 3D illustrates a two bionic lens system with a first bionic lensbeing used as an outward facing camera and a second bionic lens beingused as an outward facing projector.

FIG. 4A is a flow chart illustrating a calibration method for the bioniclens system.

FIG. 4B is an example of a calibration system for the method of FIG. 4A.

FIG. 5A is a flow chart illustrating a method to determine andcompensate for distortion in the bionic lens system.

FIG. 5B is a top cross-sectional view of the bionic lenses as oneexample of the method of FIG. 5A.

FIG. 5C is a perspective cross-sectional view of the bionic lenses asanother example of the method of FIG. 5A.

FIG. 5D is a top schematic view of two bionic lenses being used during acalibration process.

FIG. 5E illustrates an example of a bionic lens being used as anin-to-out projector during a calibration process.

FIG. 5F illustrates an example of a bionic lens being used as anout-to-in camera with the display being used as a light sensor during acalibration process.

FIG. 6 is a flow chart illustrating a method to utilize the bionic lenssystem to determine eye and lens characteristics.

FIG. 7A is a flow chart illustrating a method to utilize the bionic lenssystem to determine an interpupillary distance of two eyes.

FIG. 7B illustrates a schematic of a system utilizing two bionic lensesto determine an interpupillary distance of two eyes.

FIG. 8A is an example of utilizing the bionic lens system to determineobject characteristics.

FIG. 8B is a flow chart illustrating a method of utilizing the bioniclens system to determine object characteristics.

FIG. 9 is a flow chart illustrating another method of utilizing thebionic lens system to determine object characteristics.

FIG. 10A is a flow chart illustrating a method of determining focus orgaze information via the bionic lens system.

FIG. 10B is a schematic of a system utilizing a first bionic lens and asecond bionic lens to determine gaze information of a wearer.

FIG. 10C is a schematic of a system using light characteristics emittedby the bionic lens detected by an object to generate individualized userexperiences.

FIG. 11 is a flow chart illustrating a method to determine user specificinformation via the bionic lens system.

SPECIFICATION

The present disclosure is related to systems and methods for improvingfunctionality, sensitivity, accuracy, and user experience with bioniclenses that are worn or inserted into a user's eye. The techniquesdescribed help improve device performance, data capture, and the like.As used herein, the terms “bionic lens” and/or “bionic contact lens” aremeant to encompass both lenses that are removably inserted onto a user'seye, as well as those that are implanted or otherwise more permanentlyconnected to a user's eye.

FIG. 1A is a front view of a system 100 including an eye 102 and abionic lens 104 positioned on the cornea thereof. FIG. 1B is an explodedview of the system 100. In this example, the bionic lens 104 ispositioned on an outer surface of a cornea 122 of the eye 102,sufficient to receive and alter light that travels to and from the eyelens 110 and/or retina 112. For ease of explanation, the eye 102generally includes a cornea 122 that forms an outer surface of the eye102, the iris 106 is a diaphragm (typically is the colored area of theeye) defining a central opening forming the pupil 108. Behind the iris106 is the adjustable eye lens 110, which helps to form images on theretina 112 by focusing and collimating light. The retina 112 is formedon an interior of the eye 102 and includes photoreceptor nerve cells,rods, and cones and connects to the optical nerve 116 to send imageinformation to the user's brain. As light enters the eye 102, the iris106 contracts or expands the pupil 108 to regulate the amount of lightreaching the retina 112, then the light passes through the pupil 108 andthrough the eye or biological lens 110. The cornea 122 and lens 110focuses the light rays onto the retina 112, which then senses light andgenerates electrical impulses that are sent through the optic nerve 116to the brain. In certain instances, light that reaches the retina 112may scatter and partially reflect off the retina 112 and exit the eye102 via the eye lens 110.

In one example, the bionic lens 104 is positioned on a user's eye 102over the cornea. The bionic lens 104 may generally include variouselectrical components and sensors, depending on the desiredfunctionality and engagement with the user, these components may bepositioned onboard a substrate forming the lens (typically transparent),allowing the sensors and electrical components to be directly in contactwith the user's eye. In other examples, the bionic lens 104 may beimplanted into a user's eye, such as replacing the biological lens 110.In these instances, the bionic lens 104 may be positioned behind theiris 106, rather than in front of the iris 106 on the cornea. As can beappreciated, the variation in the position of the bionic lens 104 likelywill change characteristics and positions of the electrical componentsand sensors. As such, any discussion of a particular implementation ismeant as illustrative only and specific examples of positioning, type,size, and functionality for the bionic lens 104 are not meant to belimiting.

FIG. 2 illustrates a simplified block diagram of the bionic lens 104.With reference to FIGS. 1B and 2 , the bionic lens 104 may include adisplay 116, an image sensor 118 or camera, one or more optical elements120 (e.g., lens 120 a, lens 120 b), a power source 121, a communicationinterface 124 or component, a processing element, and/or a memorycomponent 128. The features may be integrated into the bionic lens 104or certain elements, e.g., power and processing elements, may be incommunication with the remaining components, either wirelessly or in awired manner, that is outside of the worn bionic lens 104. The structureof the bionic lens 104 and its onboard or external components may bevaried depending on the uses and functionality, as such, the descriptionof any particular embodiment is meant as illustrative only.Additionally, although only a single bionic lens 104 is shown, it shouldbe appreciated that generally the system 100 may include a bionic lens104 for each eye 102 (left and right) of the user. In these instances,the two bionic lenses 104 may be the same as one another or may bedifferent, e.g., one may include the image sensor 118 and the other mayinclude the display 116, or the like, to minimize the size requirementsfor the lenses 104. To that end, many components of the bionic lens 104may be formed as an integrated circuit and may be transparent wherepossible, allowing as much light as possible to be transmitted throughthe bionic lens 104 to the user's eye 102. As an example, in instanceswhere the electronic components are opaque, they may be arranged on thelens 104 in a manner that reduces the occluded area, i.e., have as smallof a footprint on the lens 104 as possible, such as arranged around aperipheral edge or clustered together in particular location. Thespacing and type of electronic components will also vary based on theuses of the lens 104 and included electronic components.

The image sensor 118 may be substantially any type of device that cancapture or sense light, either or both visible and non-visiblewavelengths. For example, the image sensor 118 may be a complementarymetal oxide semiconductor (CMOS) or charged couple device (CCD) camera.The display 116 may be a microdisplay or other small format display thatgenerates light output or emits light corresponding to electronicinputs, e.g., forms images or patterns. In some instances, the display116 may also be used as an image sensor, such as by utilizing lightsensitive elements, such as light emitting diodes or the like, togenerate electrical signals corresponding to the amount of lightdetected. The type of display 116 will likely depend on the sizeconstraints for the lens 104, but in some instances may be a micro lightemitting diode (LED) or organic light emitting diode display. In manyembodiments, the display 116 will be oriented to face towards the user'seye 102 when positioned on the cornea. However, in some examples, thedisplay 116 may also include outward facing elements, to generate lightthat is directed away from the user's eye and/or may include one or morelenses so that the image can be projected from the front of the bioniclens 104 as well and capture images along the user's sight line. Thedisplay may be arranged as a projector to project light out of the eyeor project light in a manner that is likely to be reflected from theretina and reflect back out of a user's eye. As such, although the termdisplay is used, the functionality may be similar to a projector in someembodiments herein.

The optical component 120, e.g., one or more lenses 120 a, 120 b, of thebionic lens 104 is a transmissive structure that focuses or disperseslight beams. Generally, the optical component 120 will be a focusinglens that is in optical communication with the eye lens 110 and thedisplay 118 to collimate light from the display 118 such that a relaxeduser's lens (focused at infinity) forms a sharp image of both theoutside world and off the display 118 onto the retina 112. In someinstances, the optical component 120 may be integrated into the displayor otherwise work with an integrated display that collimates light ontothe biological lens. The type of optical component 120 and shape, e.g.,convex, concave, etc., will vary depending on the desired images to beformed on the user's eye and applications. In some instances, theoptical element 120 may form a substrate for the remaining components ofthe bionic lens 104 and may not include any light varying aspects,merely acting as a transparent support for the electrical components ofthe bionic lens 104.

The power source 122 provides power to the components of the bionic lens104 and may be a portable or wireless source or a wired variation, e.g.,battery, magnetic or inductive coupling, radiative techniques, or thelike. The type of power source 122 and the location will depend on thepower demands for the lens 104 and can be varied as needed. Thecommunications interface 124 transmits information to and from thebionic lens 104 and between the various components of the lens 104, andmay include traced electrical connections, wireless mechanisms (e.g.,near field radio transmissions), and so on. The processing element 126controls the various components of the bionic lens 104 and is anyelectronic device capable of processing, receiving, and/or transmittinginstructions, including one or more graphics processing units, circuits,processors, or the like. The memory 128 stores electronic data used bythe processing element 126, image sensor 118, and display 116.

It should be noted that the bionic lens 104 and system can be arrangedand function in a variety of different manners, depending on the desiredapplication. FIGS. 1B-1F illustrate various ray diagrams for differentuse cases for the system 100. Methods and techniques utilizing thevarious uses shown in FIGS. 1B-1F are described in FIGS. 3A-8C. Withreference to FIG. 1B, the light rays illustrates an example of thesystem 100 where the bionic lens 104 includes an integrated display andan outward facing image sensor 118. In this example, the display 116emits light towards the user's retina 112 and the image sensor 118captures light entering the retina 112 (e.g., environmental light). Thisstructure may be used to calibrate the bionic lens, detect gaze of theuser, allow three dimensional scanning, as well as other types ofapplications or uses that require a light projection onto the user'sretain, such as alternative reality and scanned camera functionality.

FIG. 1C illustrates another implementation of the system 100illustrating the directional light rays, where the bionic lens 104includes an inward or retinal facing image sensor 118 or camera andoutward facing projector or display 116. In this example, the onboardimage sensor 118 can capture light rays as they appear or form on theretina, including retina images, and the display 116 can project lightdirectly out of the eye, such as onto objects within the environment.This implementation may be used for applications such as weareridentification, health analysis, lens alignment, as well as applicationswhere light is projected outwards, e.g., user interactive games orapplications, detection of a wearer's gaze, and three dimensional imagemapping via structured light.

FIG. 1D illustrates an implementation of the system 100 where the bioniclens 104 is used as an in-to-out projection system. In other words, thedisplay 116 is oriented so as to direct light inwards towards the retina112 such that at least some of the light rays will reflect from theretina 112 and travel back through the cornea and out into theenvironment. This type of light ray reflection and system can be used inapplications where it is desirable to project retinal reflected lightinto the environment, such as for wearer interaction with differentobjects, detecting gaze, and three dimensional aspects via structuredlight.

FIG. 1E illustrates an implementation of the system 100 where the bioniclens 104 can act as a retinal scanner. For example, the display 116 isoriented to direct light, such as a structured light pattern 125 towardsthe retina 112, then as the light reflects off of the retina 112, thelight rays extend out of the eye and onto a reflective surface 123, suchas a white board, mirror, or the like, that may scatter the integratedretina pattern back towards the bionic lens 104 so as to be captured bythe onboard image sensor 118. In this manner, the image sensor 118 maydetect an integrated light pattern 127, which can then be compared tothe input structured light projection 125 to generate a retina pattern129 that is varied based on the biological characteristics of the retina112.

FIG. 1F illustrates an implementation of the system 100 where the bioniclens 104 is configured to utilize the display 116 as a light sensor,capturing light from the environment as it is reflected off of theretina 112, e.g., the display 116 may be faced towards the retina 112but act to detect light as it reflects from the retina 112. Thisimplementation may be used in applications where imaging of the exteriorenvironment as detected by the retina 112 are desired, such as, but notlimited to, calibration techniques, gaze detection, and object threedimensional scanning.

As shown in FIGS. 1B-1F, the system 100 can be arranged to allow thebionic lens 104 and eye to can function in a few different manners,allowing the user's eyes to essentially act as a camera, act as aprojector, and/or a projector-camera system. With reference to FIG. 3A,in this example, the bionic lens 104 includes the image sensor 118 andas the user moves his or her cornea to look at an object 130, the lightof the object 130 will reflect from the object 130 and pass through thebionic lens 104 to be captured by the image sensor 118. In this example,the light may be captured by an image sensor positioned on a front orexterior surface of the bionic lens 104 and/or inwards facing display116 that is leveraged as an image sensor. Continuing with this example,the light beams that reach the eye 102 via the bionic lens 104 can betherefore be captured and in some instances, depending on the diameterand transmissivity of the bionic lens 104, certain rays may reach theuser's retina 112 around the lens 104 as well.

As the light rays exit the bionic lens 104, e.g., around the imagesensor or through a transmissive image sensor 118, the rays will befocused by the user's eye lens 110 onto the retina 112. These light raysthen form the image signals that are transferred to a user's brain,forming the image. Utilizing the captured light rays, the image sensor118 in the bionic lens 104 will generate an image corresponding to theobject 130 viewed by the user. As the user moves his or her eye 102,e.g., to look at different objects, the bionic lens 104 will generallymove therewith, allowing the image sensor 118 to capture the variousdifferent views and viewpoints of the user. As used herein, the term“eye as a camera” is meant to refer to the concept of utilizing theimage sensor 118 in the bionic lens 104 to capture images correspondingto objects and scenes as viewed by the user, such as through the userverging or moving their eyes to look at an object or scene, generallythe vergence of the eyes may be correlated to a focus point or locationof the user, i.e., where the user is focusing his or her eyes.

With reference to FIG. 3B, in this example, the system 100 may beutilized to project light out onto objects. The display 116 generatesimages that create device rays 134, these rays 134 travel from thedisplay 116, into the user's eye 102 and are focused by the eye lens 110onto the retina 112. Due to the characteristics of the retina 112, anddepending on characteristics of the device rays 134, some light rays maybe reflected off of the retina 112 forming retina reflected rays 136.These retina reflected rays 136 may travel from the retina 112 outwardstowards the eye lens 110 and through or around the bionic lens 104towards an external object. As used herein, the term “eye as aprojector” is meant to refer to the concept of generating light to bereflected off of the retina 112 and back out through the eye lens 110 toreach the bionic lens 104 and/or be transmitted therethrough or aroundthe lens 104 to the exterior environment. Typically, in these instances,the wavelengths of the projected light may be selected to be in a rangethat will be highly reflective or scattering from the retina, to allowas much of the projected light as possible to reflect from the retina112, rather than being absorbed by the retina 112. Also, the lightwavelengths may be modulated or otherwise low power to reduce damage tothe eye or be uncomfortable for the user. In these instances, themodulated light may be detected via an image sensor or other camera witha demodulation module. Additionally, modulating the light sources mayact to increase sensitivity detection in light of lower projectionbrightness from the bionic lenses and help to reduce noise (e.g.,background or ambient light).

With reference to FIG. 3C, in this example, the system 100 may be usedto both project light outwards onto objects in the environment(in-to-out projector) and include an outward facing camera or imagesensor 118 that detects light as reflected on environmental objects.This dual directional light is demonstrated by the solid and dashedlight rays being generated by the display 116, traveling through thelens 110, reflecting of the retina 112 and traveling out of the eye, andreflecting from an object back towards the bionic lens 104, which canthen use the outward facing image sensor 118 to capture images of thereflected light.

With reference to FIG. 3D, in this example, the system 200 may includetwo bionic lenses 204 a, 204 b, the first positioned on a user's firsteye 202 a (e.g., right eye) and the second positioned on a user's secondeye 202 b (e.g. left eye). The bionic lenses 204 a, 204 b may be thesame as one another or one may include an outward facing image sensorand the other may include an outward facing display or light projector.In this example, the first lens 204 a and eye 202 a act as a projectorto generate light rays by the display 116 that reflect out of the eye202 a and onto an object and the second lens 204 b and eye 204 b act asa camera to capture images of the object and the light as projected bythe eye as a projector system. As used herein, the term “eye as aprojector/camera” is meant to refer to the concept of utilizing thebionic lens 104 to both generate light reflected out of a user's eye andcapture light coming into the user's eye, either through two differentlenses (as shown in FIG. 3D) or via a single lens system (as shown inFIG. 3C), e.g., the display generates the light and the camera capturesthe light after it passes from the retina 112 back onto an externalobject. It should be noted that while the eye as a projector may beexecuted utilizing two lenses 204 a, 204 b, in some embodiments, asingle bionic lens 104 can both generate the projected light and captureimages.

Calibration Techniques

Applications utilizing a bionic lens system 100, such as alternativereality, video gaming, and informational systems, will typically requirethat the bionic lens 104 has calibrated display and image formingcomponents that account for distortions due to the optical element 120of the lens (or image sensor 116) and hardware limitations for thedisplay 116, as well as is calibrated to account for variations specificto a user's eye 102, e.g., eye lens 110 optics and the like.

FIG. 4A illustrates a method to calibrate the bionic lens 104 to adjustfor hardware and optical element 120 or other onboard optical features,such as focusing lenses including on the image sensor or display, thatmay generate distortions. The method 210 begins with operation 212 and amodel eye calibration system is generated. An example of the model eyecalibration system 222 is shown in FIG. 4B, which includes an artificialreplica of a human eye 226 that may be a single lens (or multiple lensesand apertures) selected to have an effective focal length (D1) of astandard human eye, e.g., around 17 millimeters, as well as a sensor 224or screen at a back area thereof and optionally a display that cangenerate images. The calibration system 222 is configured to be able toreceive the bionic lens 104, connected in the same manner as it would beworn on a user, e.g., on the outer surface of the eye replica 226.Utilizing the model eye system 222, the method 210 proceeds to operation214 and structured light patterns or other calibration patterns areprojected, either by the display 116 of the bionic lens or by acalibration display integrated into the calibration system 222 or inoptical communication with the system 222, e.g., that is able to becaptured by the image sensor 224 and the lens 104.

As the images are displayed, the method 210 proceeds to operation 216and calibration images are captured either by the image sensor 224 orthe calibration system 222 or directly by the onboard image sensor 118of the bionic lens 104. In embodiments where the images are captured bythe calibration system 222 image sensor 224, the sensor 224 ispositioned so as to replicate the focal length and field of view of thehuman eye to view the images through the bionic lens 104 as they wouldappear to a human user, i.e., as would be formed on the retina of theuser. It should be noted that the captured calibration images arecaptured as viewed through the bionic lens 104 and thus through theoptical element 120 (e.g., lens material) of the lens itself. The numberof calibration images may be varied as needed depending on the type ofcalibration and sensitivity for the bionic lens.

Once the calibration images are captured, the method 210 proceeds tooperation 218 and a processing element or computer determines theintrinsic parameters of the bionic lens 104, such as focal length, lensto center characteristics, distortion parameters including thedistortion introduced by the optical element 120 of the bionic lens 104.For example, the computer can compare the known characteristics of thecalibration patterns or structured light patterns (e.g., pattern 228) asdisplayed (e.g., original or input images) to the light pattern imagesas captured by the model image sensor 224. Differences between thecaptured light patterns and the displayed patterns reflect thedistortion introduced by the optical element 120 of the bionic lens 104.For example, as shown in FIG. 4B, the calibration pattern 228 ascaptured by the image sensor 224 is shown distorted in shape, ratherthan a rectangular shape of the input calibration pattern. Thedistortion characteristics introduced at various locations of the bioniclens 104 can be stored as a mapping, lookup table, or other relationalstructure. In some embodiments, the distortion characteristics can becompared against a threshold to determine if the lens quality meets aparticular standard, e.g., manufacturing or factory standards.

In some instances, utilizing the determined distortion characteristics,the method 210 proceeds to operation 220 and a lens correction map isgenerated. For example, the computer can generate a calibration map orother algorithmic relationship that can be applied to input images to beformed on the onboard display 116, that will counteract the distortionintroduced by the optical element 120, ensuring that the user will seethe desired displayed images (rather than distorted images). Utilizingthe calibration system 220, manufactures can help to ensure quality andstandardized products.

With reference to FIG. 5A, in some instances the calibration for thebionic lens 104 may also take into account distortions introduced by thebiological eye lens 110. For example, when utilizing the eye as aprojector, distortions in the light exiting the user's eye 102 (e.g.,due to the user's biological eye lens 110) may be determined andcompensated. In one example, the calibration method 230 includesoperation 232 and one or more calibration patterns (e.g., structuredlight) are displayed by the display 116 of the bionic lens 104 into theuser's eye 102. As described with respect to FIG. 3B, as the light raysfrom the display 116 scatter or reflect from the retina 112, the lightrays will pass back through the user's eye lens 110 and bionic lens 104to an external object.

As the calibration patterns are displayed and directed out of the user'seye 102, the method 230 includes capturing calibration images of thelight patterns after they have exited the eye 102. In one embodiment,shown in FIG. 5B, the exiting light patterns may be captured directlyvia an external calibration sensor 252. In this example, the calibrationsensor 252 is positioned in front of the user's eye 102 and configuredto capture all light rays at all angles as they exit the bionic lens 104and the eye 102. In some instances, the camera 252 has a sufficientlylarge lens and is focused at infinity in order to ensure that all thelight rays exiting are captured. The light patterns may also bemodulated, so that low power illumination can be used, as not to damagethe user's retina, but still allowing the modulated light to be detectedby an appropriate sensor/camera with demodulation circuitry.

In another embodiment, shown in FIG. 5C, the onboard image sensor 118can be used to capture the light rays. In this embodiment, the user ispositioned in front of a mirror 256 or other reflective orretroreflective surface and as the light rays generated by thecalibration pattern as formed on the retina 112 exit the eye 102 andbionic lens 104, they are collimated by both the user lens 110 andoptionally the optical element 120 within the bionic lens 104, and thenreflected back towards the bionic lens 104 from the surface of theretroreflector 256. As shown in FIG. 5C, the calibration distortedpattern 254 as formed on the retina and collimated by the lenses isvisible within the retroreflector 256. The bionic lens 104, via theonboard image sensor 118, then captures an image of the reflection,including the distorted calibration pattern 254.

With reference again to FIG. 5A, after the calibration images of thedistorted calibration pattern exiting the eye are captured in operation236, the method 230 proceeds to operation 238 and a processing elementor computer determines the distortion. For example, the inputcalibration patterns as displayed by the display 116 into the user's eye102 are compared to the captured distorted patterns and the variationsbetween the two images can be used to determine a distortion mappingdescribing the distortion imparted by the system 100, e.g., thebiological lens 110 and the bionic lens 104. Utilizing the distortion,the method 230 can then proceed to operation 240 and the system 100 cancompensate for the distortion, such as by generating a distortion mapthat is applied to input images to be displayed by the onboard display116, such that the images will be corrected to compensate for the systemdistortion.

It should be noted that in some embodiments, the calibration method 230may be performed utilizing non-visible light, such as infrared light orother light wavelengths that may scatter less on a user's retina 112, toallow more accurate calibration.

FIGS. 5D-FIG. 5F illustrate other calibration examples where the Inanother embodiment, as shown in FIG. 5F, the onboard image sensor 118 ofthe bionic lens 104 can be used to capture and image the light rays. Inthese examples, light rays generated by the calibration pattern arefocused by the biological lens 110 to form an image on the retina 112.This image scatters off the retina 112 and is collected by both thebiologic lens 110 and the bionic lens 104 to form an image on thedisplay 116, which in some instances is an LED array. As can beappreciated, in LEDs, even those in a display, can act as light sensors,allowing the display 116 to detect light values and act as a lightsensor array to capture the image of the calibration pattern that passesthrough both the user's biologic lens 110 and bionic lens 104. Fromimages of the calibration patterns, the intrinsic parameters of theuser's lens 110 and the bionic lens 104 can be determined.

With reference to FIG. 5D, the system 100 is arranged as aprojector-camera system in order to calibrate the bionic lenses 104and/or the combination of the biological lens 110 and the bionic lens104, i.e., compensate for distortion due to the optical elements 120 ofthe bionic lens 104, the display 116, or the like, and the biologicaldistortions of the wearer's lens 110. In this example, a firstcalibration pattern 227 is projected by the first bionic lens 104, aprojected calibration pattern 231 is formed on the projection surface256, which may be a reflective surface, such as a retroreflector ormirror, and the second bionic lens 104 captures a calibration image 229corresponding to the formed calibration pattern 231. In this example,the first lens 104 projects the image and the second lens 104 capturesthe image, but in other examples, both lenses may project and captureimages. In this example, the captured calibration pattern image iscompared to the first projected calibration image and used to determineand correct for distortions within the system.

FIG. 5E illustrates a system utilizing the eye as a projector tocalibrate or correct for distortions in the bionic lens 104 and/orbiological lens 110. In this example, the display 116 projects a firstcalibration pattern 235 directly onto the retina 112, i.e., inwardsfacing versus projecting out onto a projection surface. A calibrationpattern image 237 is formed on the retina 112 as the light reflects fromthe retina 112 and then as shown in the ray diagrams, reflects back outthrough the eye, including through both the bionic lens 104 andbiological lens 110. The reflected calibration pattern 239 is thenformed on a projection surface and now includes distortions added by thetravel through the lenses 104, 110. In one embodiment, the projectionsurface 239 includes a separate image sensor, which then captures animage of the pattern 239 as it is formed on the surface. As noted above,the light used to form the calibration pattern projected onto the retina112 may be modulated or otherwise selected to reduce power and ensure nodamage to biological tissue.

With reference to FIG. 5F, this system may be similar to FIG. 5E, butthe onboard image sensor of the bionic lens 104 is used to captured animage of the distorted calibration pattern 243. Specifically, thedisplay 116 of the bionic lens 104 generates a light patterncorresponding to an input calibration pattern 241. The input calibrationpattern 241 is projected towards the retina 112, and a reflectedcalibration pattern 245 is formed. As shown by the light directionalarrows, the image of the reflected calibration pattern 245 then travelsback through the biological lens 110 and bionic lens 104, the display116 acting as an image sensor or the onboard image sensor 118 thencaptures a calibration image 243 of the distorted pattern. The captureddistorted calibration image 243 can then be compared to the inputcalibration image 241 to determine and correct for distortions caused bythe biological and bionic lenses 110, 104.

Determining Eye and Positional Characteristics

In another example, the system 100 can be calibrated to more accuratelydetermine characteristics of the user's eye spacing and body spacing,which is helpful in generating display images that are viewpoint based,require stereo information, or otherwise include positional information,e.g., alternative reality, virtual reality, and depth overlays. FIG. 6illustrates a method 270 to determine various orientationcharacteristics utilizing the eye as a camera and/or eye as a pro-camsystems. The method 270 begins with operation 272 and a calibrationpattern or reference pattern is displayed on a surface, such as acheckerboard or other structured light pattern. For example, acalibration pattern may be displayed on an external display, such as acomputer screen, television screen, or may be a statically formed image(e.g., poster board). In another example, the calibration pattern may bedisplayed by the system 100 itself, e.g., by the onboard display 116 andrelying on the reflection of light back out of the eye 102.

As the calibration pattern is displayed on a surface, one or morecalibration images are captured by the image sensor 118 while the user'seye 102 in at a first position. For example, while the user is standingdirectly in front of the calibration pattern or is looking at thecalibration pattern from a first angle or other first location. Themethod 270 then proceeds to operation 276 and a second set ofcalibration images or image is captured by the image sensor 116 at asecond position, e.g., as the user is standing at a second locationrelative to the calibration pattern, looking at the calibration patternfrom a different angle (moving his or her head or eyes), etc. In someinstances, the position of the calibration pattern may be varied, ratherthan the user. For example, if a television screen is used, thecalibration pattern can be projected at different areas on the screen(e.g., upper right hand corner and then the lower left hand corner) orotherwise varied to allow calibration images to be captured at differentpositions.

Operations 274 and 276 may be repeated any number of times as needed,depending on the desired calibration and orientation information needed,e.g., N number of images may be captured at N number of orientations. Insome instances, calibration images may be captured at three or moredifferent positions to provide additional information that can be usedin the calibration and orientation process. Additionally, in many cases,the method 270 may include capturing two images for each position, suchas from the left and right bionic lenses 104 as the user looks at thecalibration pattern.

Utilizing the calibration images at the discrete positions, the method270 may proceed to operation 278 and a computing device, such as one ormore processors, analyzes the captured images, along with thecharacteristics of the calibration pattern (e.g., pattern size andfeatures, projected or displayed location, etc.). Generally, theanalysis will compare the input or known calibration characteristics tothe characteristics of the captured calibration pattern to determinedifferences, such as the translations, rotations, and other extrinsiccharacteristics of the images. For example, computer vision algorithmsused to calibrate stereo camera pairs can be applied to the capturedimages, and the intrinsic and extrinsic properties (poser, or locationsand rotations of the cameras where each picture was captured from) canbe determined. Utilizing these parameters and a model of a humaninterpupillary distance (or known distance), fixed separation distanceof the lens image sensors that are rotating behind their points ofprojection, the interpupillary distance and eye rotational axes can bedetermined.

With the analysis, the method 270 proceeds to operation 280 and the eyeand lens characteristics for the system 100 can be determined. Forexample, the distance between the images sensors 118 in each of thebionic lenses 104 worn by the user (e.g., left eye lens and right eyelens) can be determined, which may be correlated to the distance betweenthe user's eyes. Additionally, in instances where the calibrationpositions are varied by a user looking up, down, right and left, thechanges in perspective in the calibration images can be used todetermine the pupil rotational characteristics, e.g., pivot point orrotational axis relative to the eyeball and center of rotation for theeye. This information is useful to update images that are displayed viathe lens 104 to the user that include elements that are varied based onthe perspective. Similarly, the analysis can be used to determine thecenter of rotation of the user's head as well, given that the spacingbetween the eyes can be determined and by comparing multiple imagescaptured as the user moved his or her head into different positions.

Similarly, another method for determining interpupillary distance andeye measurements is shown in FIG. 7A and a schematic of the system isshown in FIG. 7B. With reference to FIG. 7 , the method 290 may beginwith operation 292 and the image sensors 118 of the bionic lenses 104may receive light corresponding to user reflection, e.g., light asreflected from a retroreflective surface 256 or a mirrored surface thatcorresponds to the user. In some instances, the user can followinstructions (e.g., via an application, displayed via the onboarddisplay 116 or the like), that directs the user to stand in front amirror or other reflective surface 256, allowing the lenses 104 tocapture images of the user's face. The method 290 includes capturing oneor more facial reflection images by the image sensor 118, the facialreflection images including at least the user's left and right eyes 102.

Utilizing the captured eye images, the method 290 proceeds to operation296 and a processor or computing element analyzes the captured eye imageto determine eye spacing characteristics, such as the inter-pupillarydistance. As an example, the processing element may analyze the capturedimage 251 and using the detected irises and an estimated D1 distance(e.g., a typical diameter for iris, e.g., between 10.2 to 13 mm, with anaverage size of 12 mm), and then use photogrammetry or other measurementtechniques to determine the distance between the two pupils within thecaptured image 251. Using an image detection algorithm, the location ofthe irises on in the eye image can be determined (such as by using colordetection, subtracting the white of the corner from the colored portionsof the iris), and then applying an average diameter of the iris, theprocessor can extrapolate the distance between the two pupils of theuser's eyes using photogrammetry and other image analysis techniques.

Determining External 3D Information

The system 100 can be used to provide information to the user andprograms including the bionic lens 104 functionality regarding the shapeand characteristics of objects surrounding the user, e.g., theenvironment. This information can be helpful to further tailor imagesthat will be displayed by the display 116 to the user to conform to thedetected shape of the object, providing a more realistic appearance inthe virtual space or the like.

One method for determining object shapes and topography is shown inFIGS. 8A and 8B. In this example, the method 310 begins with the lens104 projecting an input structured light pattern 301 onto an object 302.This may be done via the indirect or reflected light from the display116 or from an outward facing display and lens (outward facingprojector) that directly emits light outwards away from the user'sretina 112. In some embodiments, the display may be positioned in frontof the bionic lens 104 relative to the retina, such that the light maynot travel through the bionic lens 104 before reaching the object. Asshown, the projected light pattern 301 is projected onto an object 302,forming a distorted pattern 300 as the light pattern varies to conformto the shape of the object 302.

The method 310 then includes capturing one or more images by the imagesensor 118 of the projected light pattern 300. In one example, the imagesensor 118 of the non-projecting lens 104 in a projector-camera systemcaptures the images while the lens 104 in the other eye projects thelight pattern. In another example, the same lens that is projecting thelight may also act to capture images of the projected light on theobject 302. The captured images of the distorted light pattern are thenanalyzed by a computer or processor in operation 316. In this operation,the object topography can be determined by analyzing changes between theinput light pattern and the distorted light pattern 300 as projectedonto the object 302, e.g., changes in dots or other pattern elements asthey interact with the object surface, e.g., a planar surface may notintroduce many changes in the shape of the pattern, whereas a curvedsurface may introduce a particular distortion that corresponds to or iscomplementary to the shape of the object 302.

It should be noted that the method 310 can be done utilizing both lenses104, one projecting and one capturing images, a single lens that bothprojects and captures images, or via a dual projecting/capturing systemwhere both lenses project light patterns and both capture images of theprojected light patterns. With this last example, the light patternsemitted from the two lenses 104 may be modulated or otherwise tailoredto be identifiable as corresponding to the particular lens (e.g., bycolor, pattern element shape, projection rate, size, or the like). Inthis example, two separate patterns from slightly offset locations canbe used with known distance relationships between the origination sourceto provide further data to assist in determining environmentalcharacteristics and topography. Specifically, utilizing photogrammetryalgorithms and identifiable correspondences between the images capturedby the two offset lenses 104 (e.g., right eye and left eye), known orestimated interpupillary distance or distance between the two lenses104, a head orientation of the user, position of a user within anenvironment, object shapes and positions relative to the user can bedetermined.

Utilizing the projected light, captured distorted images, and the like,the lenses 104 can be used assist in the computation of simultaneouslocalization and mapping (SLAM) that generates/updates a map of anunknown environment while simultaneously determining and tracking theposition of the user or the lenses 104 within the environment. In someembodiments, the bionic lens may include a depth sensing camera thatutilizes techniques, such as time of flight or the like, to determinedepth characteristics of the object and/or stereogrammetery techniquesas well. These techniques can then be used to render viewpoint adjustedcontent based on the orientation of the user's head and/or a physicalorientation of the user within an environment.

In another example, the system may use the projected light from the lens104 to illuminate the environment or an object from different positions,allowing the object's features to be determined from the variations inthe light and shadows and comparing those changes across the differentlight positions. FIG. 9 illustrates a method 320 using lightingvariations to determine object shape. With reference to FIG. 9 , themethod 320 may begin with the lens 104 projecting light (either directlyor via an eye as a projector system) onto an object from a firstposition. In one example, this first position is the position of thefirst lens 104 as worn by the user, such as the right or left lens. Themethod 320 then proceeds to operation 324 and a first light image iscaptured, the first image corresponding to the illumination of theobject by a first light source, in this case the first lens 104.

The method 320 then proceeds to operation 326 and light is projectedonto the object from the second lens 104, e.g., the other of the rightor the left eye. In this manner, the light is projected from a differentangle, e.g. offset from the first image at least by the inter-pupillarydistance or other distance between the two lenses 104. In anotherexample, the user may actually tilt his or her head or body positionrelative to the object and the same lens that captured the first imagemay capture the second image from the same viewpoint, but with the lightreflecting from the object at a different position. As the light isbeing projected onto the object from the second angle, the lens 104captures a second light image.

The method 320 may then repeat the projection and capturing operations330, 332 with the light source at a third or more position. Theadditional positions may be generated by using additional light sources,such as a view from a different head position, ambient lights, mobilephone, headphones, or a wearable accessory including a light source,that projects from a location other than one of the user's eyes or byhaving the user physically tilt his or her head or eyes to anotherorientation relative to the object.

For example, the user can tilt his or her head as various images arecaptured, the titled position of the head acts to vary the angle of thebionic lenses relative to the object to define a third location. In thisexample, the captured images may need to be pre-processed before beingused to determine the object characteristics to correct for the tilt ofthe head, i.e., straighten the image to a reference frame that matchesan orientation of the other captured images. As a specific example, afirst image is captured with the object being illuminated with lightemitted from a bionic lens worn in the user's left eye with the user'shead at a first orientation, a second image is captured with the objectbeing illuminated with light emitted from a bionic lens worn in theuser's right eye with the user's head in the same first orientation, andthen a third image is captured with the user's head titled at 45 degreescounter clockwise relative to the object and being illuminated by one ofor both the right lens or the left lens. The three or more images canthen be analyzed to determine various characteristics of the object. Invarious examples, the system may also include an external sensor thattracks or determines the user's head position, allowing the system tomore easily compensate the captured images in light of the headposition, e.g., a gyroscope, inertial measurement unit, gyro, globalpositioning element, or the like. In one example, the externalpositioning sensor may be located in headphones or ear buds worn by theuser or on another head mounted module.

Utilizing the three or more light positions and corresponding lightimages, the method 320 proceeds to operation 334 and a processor orcomputer determines the shape of the object based on the differentlighting and shading characteristics in the difference light sourceconditions images. For example, the different images from the lightdirections will include brightness and angle incident variations thatcan be used to populate a lookup table or other mapping structure, todetermine normal angles of the object's surface. Using the determinednormal values from the light angles, the processor can integrate thevarious normal to output a surface value and shape.

Gaze Detection and Data Transfer

The bionic lens 104 system can be used to assist in gaze detection,which can be used in many applications to vary outputs or providedifferent user experiences based on gaze direction and orientation. Inone example, a projector-camera system can be used to include oneprojected light display, which may be a spotlight or collimated beams,and a camera in the other lens to capture the location of the projectedlight. The location of the projected light can then be correlated to thedirection of the user's gaze, since the bionic lens 104 may generallymove with movement of the user's eye 102 and head, such that as a userlooks around, the projected light beam may be moved correspondingly. Theprojected beam or spotlight may also be modulated to be specific to theuser or the specific lens 104 within the user, e.g., right lens or leftlens, to allow determination of the gaze direction for a specific eye.If a single lens is used, the gaze direction of one eye including thelens light beam may be considered to be the same as the other eye.

Additionally, a gaze or focus point for the user's eyes can beestimated. FIG. 10A illustrates a method to determine a focus location,the method 400 includes activating a light source from a bionic lens104, either from the bionic lens' display, imaged onto and reflected offthe retina, then projected into space by the user's lens, and/or from afront facing display and lens (projector) on the bionic lens 104. Thelight source may be modulated, colored, or otherwise identifiable ascorresponding to a particular lens 104 such that the light source can bematched to a particular eye of the user. In some examples, the userwears a bionic lens 104 in each eye and the two lenses 104 emit amodulated light, allowing a first light source to be identified ascorresponding to the first eye and a second light source to beidentified as corresponding to the second eye. For example, as shown inFIG. 10B, a first light source 255 a is projected by a first bionic lens104 and a second light source 255 b is projected by a second bioniclens, the light sources 255 a, 255 b emitting light that reflects onto asurface 253.

The method 400 then proceeds to operation 404 and the projected lightbeams are detected. For example, external image sensors or light sensorsmay be used in certain environments that detect the light beams anddetermine the location of the projected beam, e.g., location of lightspots 255 a, 255 b on surface 253 (shown in FIG. 10C) such as an objectonto which the beam is reflected, correlating that location to the gazelocation. In some instances, the more general gaze area may be used andthe method 400 may terminate. However, in some instances a more specificfocus area may be desired. In these instances, the method 400 proceedsto operation 406 and a beam crossing location, e.g., location 257, forthe left eye lens 104 beam and the right eye lens 104 beam isdetermined. For example, the crossing point of the two light beams canbe determined via beam tracing in the 3D space surrounding the user andgeometric analysis can be used to determine the cross point. Inparticular, as shown in FIG. 10B, the lights 255 a, 255 b are tracedfrom the known projection point of the bionic lenses 104 by identifyingthe characteristics of the two lights 255 a, 255 b, their location onthe object 253, and tracing the beams back to determine where the beamscross 257.

The beam crossing point can be determined to be at a particular locationin 3D space and/or object in the user environment and estimated as beingthe focus point for the user. For example, the convergence point 257 ofthe beams 255 a, 255 b in FIG. 10B may be correlated to a focus locationof the user. The focus or gaze information in operation 408 may beoutput to a corresponding user device, computer, other element thatutilizes the gaze information to provide an interactive experience forthe user or vary the information and images displayed or output to theuser, either in the environment or otherwise. The gaze and focusinformation can be used to generate user specific outputs, such as in animmersive interactive environment, certain objects may be “smart” andinteract with the user as the user is focused on the object (asdetermined by the method 400). In this example, the object could displaya user specific output (e.g., happy birthday Bob!), light up in aspecific color, or generate another type of individualized experiencefor the user. That is, the light beams may be modulated or otherwiseinclude user data that can be detected by objects within theenvironment.

More specifically, the light beams generated and projected by the bioniclens 104 may include data, the light beams can be modulated similarly toa carrier wave with data overlaid, to transmit data between the lens 104and other computing devices, such as smart objects, or the like. Thedata could them be transmitted from a user device (e.g., smartphone) incommunication with the bionic lenses 104 to another computing device,such as a smart object, another user phone or computing device, or thelike. In some instances, the exchange of data via the bionic lens 104may be determined based on a detected gaze or focus location, e.g.,focusing on a particular object (detectable via the method 400 of FIG.10A) provides an input to the system that triggers the release ofcertain data via the modulated light beams from the bionic lens 104.

For example, FIG. 10D illustrates a system where an object 302 receiveslight from a first user and a second user, where the first user projectsa first light pattern 307 having a first identifying set ofcharacteristics (e.g., modulated light) and the second user projects asecond light pattern 309 have another different set of characteristics(e.g., modulated light) that allows the object 302 or other element todetermine the differences between light from the first user's bioniclens and the second user's bionic lens. In one embodiment, the firstlight pattern has a first emitting pattern and the second light patternhas a second emitting pattern different from the first one. Additionallyor alternatively, the light wavelengths may be different from oneanother to also distinguish between the two users.

In other instances, the bionic lens 104 and the light projected eitherdirectly outwards or via the reflection from the user's retina 112, canbe used to identify a user or gaze information or allow auxiliarydevices to more easily detect the user's eye location. For example, thebionic lens 104 may act to “glow” or illuminate the pupil (with eithervisible or invisible wavelengths) and the light, which may be moreeasily detected than a pupil location, can be tracked by auxiliarydevices, allowing a more accurate and simplified gaze tracking. This canbe used to assist in increasing accuracy for performance capture forcomputer animation and other techniques. Current performance trackingtechniques that convert a person's physical motions into animated motionmay not accurately or easily capture the person's eye movements.Utilizing the emitted glow or light from the bionic lens 104, systemscan identify and track the motion and movements of the person's eyes,allowing this motion to be more easily converted to the computeranimation realm.

Health Detection and Identification

The bionic lens 104 can be used to detect certain user characteristicsthat can be used to allow proper identification of the lens with theuser (e.g., correct lenses for the particular person and/or correct lensfor the correct eye, right or left). In some instances, the bionic lens104 can capture data corresponding to the user's eye and use thebiometric information to validate the operation of the lenses and/orprovide errors or alerts in the event that the lenses are interestedinto the eyes of a different user or in the wrong eye for the specificlens 104.

One example of biometric information that may be used is a retinal scan.FIG. 11 illustrates a method 420 to generate a biometric map for theuser's eye utilizing the bionic lens 104. The method 420 includesactivating a pixel illumination pattern to be displayed by the display116 of the lens 104. The pixel illumination pattern may include a scantype of pattern, where rows of pixels are sequentially illuminated,grayscale patterns, or may be another type of pixel or group of pixelsilluminated pattern. As the pattern is activated sequentially, the imagesensor 118 or an external sensor captures images of the retina 112(either directly or via a reflective surface, such as a mirror or whitesurface), such that the captured retina images correspond to aparticular point in the pattern, e.g., first row, first pixelilluminated, or the like.

Utilizing the captured images, along with the known characteristics ofthe pattern and its sequential illumination, the method 420 proceeds tooperation 426 and a processor or other computing element generates aneye or retinal map. The retinal map may include a correspondence ofbright or dark spots or average light reflected from the retina atvarious points during the pattern's illumination sequence. Due to thedifferent retinal structures, veins, and the like, the light reflectancefrom the retina may be different or unique for each user, allowing theretinal map to be uniquely generated for each user and each user'sspecific eye, e.g., left eye or right eye.

Alternatively, an image of the retina may be captured in a single imageusing the onboard sensors of the bionic lens 104. For example, externallights may illuminate the u retina 112, causing the light to reflect offthe retina, the reflected retinal light is then imaged onto the LEDarray by the biological lens 110 and bionic lens 104, with the LED arrayor other display 116 configured to sense light rather than emit light.In other embodiments, an inward facing image sensor of the bionic lensmay capture an image of the retina directly.

In some instances, the retina map may be compared to historical maps,such as in operation 428. In these instances, the historical retinal mapmay be compared to a current retinal map to determine if the lens 104 isinserted into the proper eye and/or if the lens 104 is being worn by thedesignated user. As another example, the retinal maps can be used todetermine if the bionic lens 104 is inserted properly, as the light fromthe display 116 may reflect from the retina 112 differently, generatingslightly different retinal maps, based on the positon of the lens 104relative to the cornea.

The method 420 may also include operation 430 where a user output isprovide based on the comparison in operation 428. For example, the lens104 may display a green light or other image that corresponds to avalidated comparison, e.g., the bionic lens 104 is inserted into thecorrect eye or correct user. In other examples, the comparison can beused to determine health information for the user, e.g., detect changesin veins or other structures within the user's eye. In these instances,the user output may include a display regarding health information orwarnings, e.g., health alerts regarding possible hemorrhages, oxygenlevels, pulse, blood pressure, blood alcohol content, and the like. Thecomparison can also be used to determine whether the bionic lens 104 hasbeen inserted onto the cornea in the correct orientation or position,such as by comparing the locations of certain retinal features in thecurrent scan as compared to historical scans. The user output may theninclude specific orientation adjustments, e.g., move lens up and to theright, etc. In some embodiments, the biometric information detected orcaptured via method 420 can also be used as part of the data transferredvia the light modulation to identify specific users.

CONCLUSION

The methods and systems are described herein with reference to certainapplications for bionic contact lenses. However, these techniques areequally applicable to other types applications utilizing displays orsensors inserted onto a user's eye. In methodologies directly orindirectly set forth herein, various steps and operations are describedin one possible order of operation but those skilled in the art willrecognize the steps and operation may be rearranged, replaced oreliminated without necessarily departing from the spirit and scope ofthe present invention. It is intended that all matter contained in theabove description or shown in the accompanying drawings shall beinterpreted as illustrative only and not limiting. Changes in detail orstructure may be made without departing from the spirit of the inventionas defined in the appended claims.

What is claimed is:
 1. A method to determine eye characteristics of auser wearing one or more bionic lenses comprising: capturing by a firstbionic lens worn in a first eye of the user a first image correspondingto a calibration pattern positioned at a first relative orientation ofthe calibration pattern to the user; capturing by the first bionic lensa second image corresponding to the calibration pattern positioned at asecond relative orientation of the calibration pattern to the user; andanalyzing the first image and the second image by a processor todetermine the eye characteristics of the user by: comparing a knowncharacteristic of the calibration pattern to a characteristic of arepresentation of the calibration pattern in the first image and thesecond image; and determining a difference between the knowncharacteristic and the characteristic of the representation of thecalibration pattern.
 2. A method to determine eye characteristics of auser wearing one or more bionic lenses comprising: capturing by a firstbionic lens worn in a first eye of the user a first image correspondingto a calibration pattern positioned at a first relative orientation ofthe calibration pattern to the user; capturing by the first bionic lensa second image corresponding to the calibration pattern positioned at asecond relative orientation of the calibration pattern to the user;analyzing the first image and the second image by a processor todetermine the eye characteristics of the user; capturing by a secondbionic lens worn in a second eye of the user a third image correspondingto the calibration pattern positioned at the first relative orientationto the user; capturing by the second bionic lens a fourth imagecorresponding to the calibration pattern positioned at the secondrelative orientation to the user; and analyzing the first image, thesecond image, the third image, and the forth image to determine an eyespacing between the first eye of the user and the second eye of theuser.
 3. The method of claim 2, wherein analyzing the first image, thesecond image, the third image, and so on up to an Nth image to determinean eye spacing includes estimating a camera distance between the firstbionic lens and the second bionic lens.
 4. The method of claim 1,wherein the first orientation corresponds to the user looking in a firstdirection toward the calibration pattern and the second orientationcorresponds to the user looking in a second direction toward thecalibration pattern.
 5. The method of claim 1, wherein the firstorientation corresponds to a first location of the calibration patternrelative to the user and the second orientation corresponds to a secondlocation of the calibration pattern relative to the user.
 6. The methodof claim 5, wherein: the calibration pattern is displayed on a display;the first location of the calibration pattern corresponds to a firstportion of the display; and the second location of the calibrationpattern corresponds to a second portion of the display.
 7. The method ofclaim 1, wherein the bionic lens comprises an onboard display, and themethod further comprises displaying the calibration pattern on theonboard display, wherein one of the first image or the second imageincludes a light reflected from the first eye.
 8. The method of claim 1,wherein the characteristic of the representation is at least one of atranslation, a rotation, or an extrinsic characteristic of the firstimage or the second image.
 9. The method of claim 1, further comprisinganalyzing the difference between the known characteristic and thecharacteristic of the representation of the calibration pattern todetermine that a quality of the bionic lens satisfies a manufacturingstandard.
 10. The method of claim 2, wherein analyzing the first imageand the second image comprises analyzing a model of a humaninterpupillary distance between the first eye of the user and the secondeye of the user.
 11. The method of claim 1, wherein the eyecharacteristics comprise at least one of a pivot point, a rotationalaxis of the bionic lens relative to the first eye, or a center ofrotation of the first eye.
 12. The method of claim 1, further comprisinganalyzing the first image and the second image to determine a center ofrotation of a head of the user.
 13. The method of claim 1, wherein thefirst bionic lens includes an integral image sensor configured tocapture the first image and the second image.
 14. The method of claim 1,wherein the calibration pattern comprises a structured light pattern.15. A system to determine eye characteristics of a user wearing one ormore bionic lenses comprising: a first bionic lens configured to be wornin a first eye of the user; and a processing element in communicationwith the first bionic lens, wherein the processing element is configuredto: capture by the first bionic lens a first image corresponding to acalibration pattern positioned at a first relative orientation of thecalibration pattern to the user, capture by the first bionic lens asecond image corresponding to the calibration pattern positioned at asecond relative orientation of the calibration pattern to the user, andanalyze the first image and the second image to determine the eyecharacteristics of the user by: comparing a known characteristic of thecalibration pattern to a characteristic of a representation of thecalibration pattern in the first image and the second image; anddetermining a difference between the known characteristic and thecharacteristic of the representation of the calibration pattern.
 16. Thesystem of claim 15, further comprising: a second bionic lens configuredto be worn in a second eye of the user, wherein the processing elementis further configured to: capture by the second bionic lens a thirdimage corresponding to the calibration pattern positioned at the firstrelative orientation to the user, capture by second bionic lens a fourthimage corresponding to the calibration pattern positioned at the secondrelative orientation to the user, and analyze the first image, thesecond image, the third image, and the forth image to determine an eyespacing between the first eye of the user and the second eye of theuser.
 17. The system of claim 16, wherein analyzing the first image, thesecond image, the third image, and so on up to an Nth image to determinean eye spacing includes estimating a camera distance between the firstbionic lens and the second bionic lens.
 18. The system of claim 1,wherein the first orientation corresponds to the user looking in a firstdirection toward the calibration pattern and the second orientationcorresponds to the user looking in a second direction toward thecalibration pattern.
 19. The method of claim 1, wherein the firstorientation corresponds to a first location of the calibration patternrelative to the user and the second orientation corresponds to a secondlocation of the calibration pattern relative to the user.